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Abstract 

This article reports the findings of a study of the Penn 
State Cooperative Extension program evaluation Web site. The 
study was performed to investigate whether a Web site could 
be used as a vehicle for organizational learning in evaluation, 
and, if learning does take place, what aspects of the site make 
learning possible. Our research suggests that a Web site can 
indeed make such learning possible, a result leading to an 
emerging culture of evaluation in an organization. To accom¬ 
plish these ends, however, the creation of a Web site should be 
approached as an intentional scholarly activity that is informed 
by theory and driven by organizational leadership. 

Introduction 

I n an era of heightened competition for scarce public and 
private resources and increased accountability, public uni¬ 
versities must demonstrate the impact of their work. Legislators, 
stakeholders, and funders are interested in the difference university 
programs can make for citizens, businesses, and communities. To 
achieve a significant competitive advantage in this environment, 
it is necessary for a university to develop the organizational 
capacity to conduct practical program evaluations, to document 
program impact, and to gain insight as to how the programs 
should be improved, or even discontinued, to free scarce 
resources. Yet organizational capacity to carry out practical, cost- 
effective program evaluation grounded in evaluation theory with 
appropriate methods is relatively underdeveloped (Stockdill, 
Baizerman, and Compton 2002). 

Program evaluation has been important in Perm State 
Cooperative Extension, that part of the land-grant university that 
conducts research-based educational programs in sixty-seven 
counties. Penn State Cooperative Extension has developed the 
capacity of many educators (faculty at the university and field- 
based in the counties) to evaluate educational programs, but some 
educators still lack the commitment. Many extension administrators 
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(state, regional, and county) perceive the value of evaluation and 
encourage it, but some administrators do not actively encourage it, 
perceiving evaluation as a drain on scarce resources and staff time. 

To expand organizational capacity in evaluation, the Penn 
State Cooperative Extension leadership and the state program eval¬ 
uator created a Web site in 2001 with different expectations for 

educators, administrators, and 
the program evaluator. For 
extension administrators, the 
Web site would convey the 
same cultural message to 
everyone, that commitment to 
evaluation is important, and it 
would create opportunities for 
individual and collective 
learning, thus providing a 
mechanism to strengthen the 
value and culture of evaluation 
in the organization. For exten¬ 
sion educators, the Web site 
would provide practical infor¬ 
mation on evaluation strategies 
appropriate for extension, thereby supporting the organizational 
commitment to evaluation. For the extension evaluator, the Web 
site would provide an efficient mechanism to motivate and educate 
almost four hundred extension educators and administrators at the 
same time with the same message. 

This article reports the findings of a study that investigated 
whether this Web site could be used as a vehicle for organizational 
learning in evaluation. We first provide a description of the site 
and how it was marketed within the organization. We then 
describe the study in which we explored the experiences of the 
educators and administrators to find out what it is about the site 
that makes learning about evaluation possible. We conducted this 
study in the tradition of reflective practice to take stock of the 
Web site after a period of three years so it could be strengthened 
in light of the expectations outlined above (Schon 1983). 

The Web site 

Change in organizations is grounded in individual learning 
(Smith 1996) that requires organizational mechanisms that system¬ 
atically allow learning to occur in a cumulative process over time 


“For extension educators, 
the Web site would provide 
practical information on 
evaluation strategies 
appropriate for extension, 
thereby supporting the 
organizational commitment 
to evaluation. ” 
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by individuals and across individuals (Argyris and Schon 1996). 
The Penn State Cooperative Extension program evaluation Web 
site is such a mechanism and can be found at <http://www. exten- 
sion.psu.edu/evaluation/>. 

The site is designed to enable educators and administrators in 
extension to achieve five specific objectives. 

• Learn valid, scientific principles of current social science 
research 

• Broaden use of data collection methods beyond surveys 

• Draw on the practical examples of evaluation on the site as 
models, irrespective of the program used in an example 

• Use evaluation data in diverse ways to demonstrate impact to 
stakeholders to improve a program, and to achieve profes¬ 
sional advancement 

• Develop an appreciation for the personal and organizational 
benefits of evaluation and for the ease of integrating evalua¬ 
tion into programming. 

The site is based on principles of empowerment evaluation, 
adult learning theories, and Web design, each of which suggested 
different features. Principles of empowerment evaluation foster 
self-determination, which suggested a need to create a Web site 
where learners felt free to reflect, learn, experiment, and improve 
themselves and their programs (Fetterman, Kaftarian, and 
Wandersman 1996). Principles of adult learning theory suggested a 
need for a range of teaching devices to appeal to an array of 
learners and to sustain attention (Kolb 1984). One such theory, 
constructionism, advocates active participation in the learning 
process so that learners do not simply acquire information but 
reconfigure it into some other way of conceiving of phenomena 
(Hawkins and Pea 1987). Constructionism suggested the need to 
take scientific principles and apply them in actual evaluation 
examples. Principles of Web design suggested a need to establish 
a site that offered quick access and sustainability—that is, a site 
unencumbered by elaborate design that required excessive com¬ 
puter memory (Nielsen 2000). 

The Web site provides information in three categories: tip- 
sheets, PA examples (Pennsylvania examples), and other 
resources. Tipsheets demonstrate the social science principles of 
evaluation, in contrast to Web sites that present social science 
principles like sections of a textbook without illustration. Each 



124 Journal of Higher Education Outreach and Engagement 


social science principle is presented in the form of a question that 
extension educators would pose as they design a valid evaluation, 
reflecting the perspective of the learner (Kolb 1984). The question 
is answered by weaving the principle into an extension illustra¬ 
tion so the learner is engaged in applying the principle to an 
extension program, reflecting constructionism (Hawkins and Pea 
1987). Positive and negative aspects of the illustration are often 
given so the learner becomes aware of the various biases associ¬ 
ated with social science methodology (Fetterman, Kaftarian, and 
Wandersman 1996). To encour¬ 
age the reading of tipsheets, 
they are short, graphically 
interesting, and informal 
(Fetterman, Kaftarian, and Wan¬ 
dersman 1996; Nielsen 2000). 

Over seventy tipsheets are now 
on the Web site, and a new one 
is added every other month. 

The second category of 
information in the Web site, 

PA examples, provides actual 
program evaluations designed 
by educators in Pennsylvania 
and reflects the perspective of the learner (Kolb 1984). Each PA 
example gives the program’s objectives, delivery method, and 
evaluation strategy; one type of example is an invitation letter 
and questions for a focus group. PA examples are designed to 
encourage a broader use of different data collection methods, 
especially underused methods like observation and experiential 
evaluation, which can reduce respondent burden but are not well 
known. To continue the instruction of social science principles, 
the strengths and limitations of the research design and data col¬ 
lection method are interwoven into the PA example, so the learner 
knows the biases associated with social science methodology 
(Fetterman, Kaftarian, and Wandersman 1996; Hawkins and Pea 1987). 
Over twenty PA examples are now on the Web site, and a new one 
is added every other month. 

The third category of information in the Web site, other 
resources, provides links to over sixty other evaluation Web sites 
across the United States, offering another opportunity to expand 
skills (Kolb 1984). Some Web sites concentrate on the evaluation 
of programs, while others present social science principles. 


“Each social science prin¬ 
ciple is presented in the 
form of a question that 
extension educators would 
pose as they design a valid 
evaluation, reflecting the 
perspective of the learner. ” 
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The Web site is continually updated to respond to ongoing 
evaluation needs; it thus remains a work in progress rather than a 
complete academic compendium on evaluation. 

Marketing the Web site 

To accomplish the short-term goal of establishing interest in 
the Web site, we first demonstrated the site online to administrators, 
indicating how it could help in their role of advocating evaluation. 
We promoted the site to educators through e-mail, sending a hot 
li nk to the site. In both demonstration and promotion, we described 
the categories of information in the site, asked participants to book¬ 
mark it, and emphasized that it would be updated regularly. 
Additionally, we conducted thirty-minute workshops statewide to 
demonstrate the Web site’s breadth of information and simple nav¬ 
igation, and distributed business cards with the site’s URL. 

To accomplish the long-term goal of sustaining use of the 
Web site, we currently conduct a hands-on workshop at new staff 
orientation two to three times a year; e-mail an announcement of 
newly posted tipsheets or PA examples to everyone in cooperative 
extension; have a feature of the month on the home page that con¬ 
nects learners to the latest posting; and distribute tipsheets and 
PA examples as handouts at evaluation workshops and in consult¬ 
ing on specific projects. 

Methodology 

The purpose of this study was to investigate whether a Web 
site could be used as a vehicle for organizational learning in eval¬ 
uation and to find out from educators and administrators what 
aspects of the site make learning about evaluation possible. 

Given the purpose of the study, we used a telephone interview 
to collect information from educators and administrators. A tele¬ 
phone interview offered the opportunity to ask qualitative, open- 
ended questions to explore the respondents’ use of the Web site; to 
probe for examples to validate the respondents’ claims about their 
use of the site; and to collect data in a less burdensome way from 
busy respondents (Salmen 1989; Patton 2002). 

The principal investigators conducted forty telephone inter¬ 
views using a purposive sample of individuals who had a range 
of experience in program evaluation (Patton 2002). We asked 
extension’s state administrators for potential participants, and 
their list correlated in all cases with ours. We interviewed twenty 
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educators (field-based and faculty). Because of the limited number 
of state and regional administrators and the potential significance 
of their leadership in evaluation, we also interviewed all ten of 
these administrators as well as ten county administrators. 

The interview questions focused on how in the last three 
years the respondent may have used the Web site in the role of 


educator or administrator; how 


“Administrators con¬ 
firmed that educators 
used a variety of infor¬ 
mation on the site, 
having seen it in recent 
promotion dossiers, 
grant proposals, and 
annual reports. ” 


the site may have encouraged the 
use of evaluation; how the site 
could be changed to help the 
respondent; and how the site 
could be changed to encourage 
others to conduct evaluation. 
The request to participate in the 
study was sent by e-mail with an 
implied consent form approved 
by the Penn State Institutional 
Review Board. The e-mail 
described the rationale for the 
study and selection of partici¬ 
pants and provided two potential 
time slots for an interview. 


Findings 

In a qualitative study with a purposive sample, the findings 
comprise the range of ideas about central themes in the data 
rather than percentages representative of the sample (Patton 
2002). In this study, three themes emerged from the interviews: 
how the respondents use the Web site, the factors that affected 
their use, and the future of the site. 

Use of the Web site: Some educators and administrators turned to 
the Web site for capacity building in evaluation; others did not. 
Interviewees provided evidence for their use of the Web site 
when they discussed the kind of information they sought, their 
reasons for using the site, and the consequences. 

Kind of information: Educators sought out a range of infor¬ 
mation on the site; they valued its scope and diversity because it 
led to learning about different evaluation approaches and methods. 
They used PA examples and tipsheets regularly and extensively. 
Administrators confirmed that educators used a variety of infor¬ 
mation on the site, having seen it in recent promotion dossiers, 
grant proposals, and annual reports. Administrators also found 
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that the educators who were most reluctant about evaluation 
found the PA examples helpful because they illustrated an entire 
evaluation strategy, making their work more efficient (e.g., a PA 
example makes evaluation “much easier” and “saves time— 
that’s huge”). 

Reasons for use: Educators’ and administrators’ responses 
reflected numerous and varied reasons for their use of the Web site. 

Educators, despite differing levels of evaluation expertise, 
regularly used the Web site: 

• to design and implement evaluations for their own extension 
programs; 

• as a general learning resource to stimulate their thinking and to 
broaden their knowledge of evaluation approaches, methods, 
and techniques, even with no evaluation project immediately 
pending. Indeed, one field-based educator reported using the 
Web site home page as her personal home page, so that every 
time she logged on to her computer she was reminded of her 
need to increase her knowledge of evaluation; 

• to design and implement evaluations to meet organizational 
expectations and values associated with promotion, annual 
performance review, and accountability reporting; 

• to design valid evaluation questions to strengthen grant pro¬ 
posals and enhance the credibility of Penn State Cooperative 
Extension with government officials and other funders in the 
community; 

• to refer evaluation approaches to their colleagues—peers, 
mentees, and interns. 

Field-based educators in particular, used the site: 

• in discussions about evaluation within groups of extension 
educators with a common programmatic interest, such as 
horticulture or family and consumer sciences. Reflecting col¬ 
lective learning, not just individual learning, these program 
groups used the site as a stimulus to do evaluation and as a 
technical resource; 

• outside their jobs: for example, in graduate school to meet 
course requirements, conduct research projects, and provide 
a resource to classmates and teachers. 

Administrators used the Web site for proactive efforts to social¬ 
ize educators to the importance of evaluation and the overall content 
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and benefits of the site at group settings and on the individual 
level. Administrators also steered educators to specific content on 
the site to help them evaluate their programs more effectively. 

Both administrators and field-based educators used the Web 
site with external community partners to introduce them to the con¬ 
tent and benefits of the site and to build their evaluation capacity. 
Some partners have become community leaders in evaluation, now 
teaching others in the community how to evaluate programs. 

Consequences of use: Use of the Web site has had identifiable 
consequences. First, the educators indicated that the site had an 
effect on them personally (e.g., “The Web site has relieved appre¬ 
hension and anxiety about doing program evaluation”). The 
administrators confirmed the effect on educators, noting increased 
self-confidence in conducting evaluation and an enhanced reputa¬ 
tion when the educators have had their evaluation posted on the site 
as a PA example. Second, both educators and administrators indi¬ 
cated that the quality and standards of evaluation improved as a 
consequence of the site, pointing to better annual regional reports, 
promotion dossiers, and external grants. And third, both educators 
and administrators noted that the site contributed to a culture of 
evaluation that is emerging within Penn State Cooperative 
Extension. However, they also noted that other factors besides the 
Web site facilitated a culture of evaluation. Those factors include 
the organizational expectations about the number and quality of 
evaluations that extension educators need to conduct annually and 
for promotion; provision of organizational resources to support an 
evaluator to provide evaluation workshops and consultation in 
addition to the Web site; and finally, the inclusive philosophy of 
evaluation accepted within the organization. This philosophy 
accommodates diverse evaluation strategies and methods, and pro¬ 
vides educators with many evaluation options and thus flexibility, 
making it possible for educators with diverse expertise and needs 
to participate in evaluation. 

Factors that affected use of the Web site: Some factors encour¬ 
aged use and others hindered it. 

The first factor that educators and administrators identified 
that encouraged use was the site itself and, in particular, a number 
of specific features. 

• Its user-friendliness and ease of navigation 

• Its comprehensiveness (e.g., “The Web site has encouraged 



Can a Web Site Be Used as a Vehicle for Organizational Learning 129 


me to use forms of evaluation I never would have thought of’ 
and “The Web site has encouraged me to explore evaluation 
more than I would have.”) 

• Its writing style (e.g., “It’s easily read.”) 

• Its accessibility (e.g., “The Web site provides easy access to 
information that I could not have gotten otherwise” and 
“Prior to the Web site, I had no idea where to go for informa¬ 
tion in evaluation. I would not have been able to do evalua¬ 
tion without the Web site.”) 

• Its message that it reflects organizational expectations and 
standards for evaluation (e.g., the site “has raised my view of 
evaluation as a value and norm in the organization” and “It 
has improved the quality of evaluation.”) 

• Its content, which is considered both relevant (e.g., “timely,” 
“important,” “practical”) and intellectually accessible (e.g., 
“It is helpful for people who have not studied evaluation” and 
the information is “idiot proof.”) 

• The tipsheets, which convey an “upbeat” message, and the PA 
examples (e.g., of the Master Gardener Program), which con¬ 
vey that “evaluation is doable.” 

The second factor that encouraged use of the Web site came 
from outside the site, the stimulus and motivation that educators 
and administrators received at evaluation workshops and from 
the e-mail announcement of new postings on the site. 

Educators noted some factors that hinder their use of the site: 
first, their lack of time to do evaluation; second, the belief some 
hold that evaluation is not important; third, their “programming,” 
which some say is a series of events, not a program; and fourth, 
their learning styles (e.g., some prefer to ask peers for ready-made 
evaluations in a program area, and others avoid the Web in general). 

Future of the Web site 

At the Web site level, educators and administrators recom¬ 
mended that we continue specific aspects of the site, such as the 
upbeat tone; the simple, succinct presentation style; the naviga¬ 
tional ease of the site; and features on new evaluation issues and 
strategies (e.g., how the Internet can help in conducting program 
evaluation). They recommended we expand the type of programs 
in the PA examples and feature evaluation strategies pertinent for 
new organizational initiatives (e.g., collaborative partnerships; cost 
recovery; Internet delivery of educational materials and programs; 
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and grant and contract acquisition) and for ongoing programs 
(e.g., youth). At the organizational level, educators and adminis¬ 
trators suggested we continue to market the site, making it even 
more visible within Penn State Cooperative Extension, in the uni¬ 
versity, and beyond. 

Discussion 

In this study we investigated whether a Web site could be 
used as a vehicle for organizational learning in evaluation, and, if 
learning occurred, what aspects of the Web site make learning 
evaluation possible. 

The findings suggest that the educators are using the Web site 
in ways that indicate it meets the five original objectives. They 
find the following objectives are met: scientific principles related 
to evaluation are intellectually accessible and applicable to their 
work (e.g., they say they are creating better questions); practical 
examples on the site are used as models; use of data collection 
methods (e.g., observation) is broader than the use of the survey; 
evaluation data are used in diverse ways, some expected and others 
unexpected (e.g., to enhance the credibility of Penn State with 
external community partners); and finally, perception that per¬ 
sonal benefits (e.g., reducing anxiety about evaluation), as well 
as organizational benefits (e.g., improvement in the quality of 
evaluation and a change in the culture of evaluation), result from 
using the Web site. 

The findings also suggest that the administrators on the state, 
regional, and county levels had a strong leadership role in augment¬ 
ing the use of the Web by educators, reflecting the administrators’ 
commitment to evaluation. Administrators not only advocated use 
of the Web site; they were intimately aware of the information on 
the site and directed educators to specific types of information rel¬ 
evant to their particular needs in program evaluation. Administra¬ 
tors were even aware of the site’s features that helped educators 
who were most reluctant to evaluate programs. These findings 
suggest that the administrators’ commitment to the Web site for 
capacity building in evaluation has contributed to the culture of 
evaluation. 

The construction and design of this Web site were motivated 
by theories of empowerment evaluation, adult learning, and Web 
design. Educators and administrators identified features of the 
site that make learning evaluation possible, and these features are 
related to the theories. For example, educators found the site to 
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embody three concepts central to the theory of empowerment 
evaluation: it is useful for learning general principles even when 
no evaluation is pending; it presents choices for evaluation 
approaches and methods; and it builds evaluation capacity of 
community organizations (Fetterman, Kaftarian, and Wandersman 
1996). Educators found the exam¬ 
ples on the site useful, which 
accords with constructionism 
(Hawkins and Pea 1987; Kolb 1984). 

Educators also found the site easy 
to navigate, which accords with 
usability theory of Web design 
(Nielsen 2000). The theories pre¬ 
dicted certain outcomes, and these 
have been realized in use of the 
Web site. These findings suggest 
that a Web site can be used as a 
vehicle for learning and applying 
evaluation skills, if the site is constructed with consideration of 
theoretical principles associated with empowerment evaluation, 
adult learning, and Web design. 

The findings also demonstrate that some educators fail to use 
the Web site due to their lack of interest in evaluation, their own 
learning styles, or the Web site’s lack of visibility within the 
organization. They did not, however, indicate that any deficiencies 
in the site accounted for their not using it. In response to the lack 
of use by some educators, we piloted a new marketing strategy 
for all new educators in the organization. At their orientation, we 
engaged these educators, each at a separate computer, in a thirty- 
minute interactive session on the Web site to find specific infor¬ 
mation about evaluation. This session stimulated interest not only in 
the site but also in evaluation. We plan to continue this marketing 
strategy. 

Three aspects of the methodology could have created bias in 
responses. The first is that the interviewers knew the participants, 
which could have led to socially desirable answers. To minimize 
this bias, in our analysis we used only interview data from 
respondents who provided concrete details that backed up their 
opinions of the Web site. A second aspect of the methodology, the 
subject of evaluation, could also have produced socially desirable 
answers. The evidence from the interviews, however, demon¬ 
strated that the respondents provided a full range of views (e.g., 


“Administrators were 
even aware of the site’s 
features that helped 
educators who were 
most reluctant to 
evaluate programs. ” 
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on use and nonuse of the site and on their acceptance and their 
aversion to evaluation). A third aspect, the purposive sample in 
the study, could have biased the findings because the educators 
and administrators in the sample may not be representative of 
those in the extension organization as a whole. In this study, how¬ 
ever, we were not concerned with the prevalence of the use of the 
Web site; that will be the focus of a future study. The concern for 
this study was the viability of the site as a vehicle for educators 
to learn and adopt evaluation skills and for organizational 
change, a scope of inquiry for which a purposive sample is appro¬ 
priate (Patton 2002). 

Despite the study limitations, this study has important 
strengths. It is one of the few studies of a Web site to use in-depth 
qualitative interviews. In using this methodology, we learned of 
the experiences of site users and nonusers that explain and validate 
their attitudes about the site (Patton 2002). We also learned about 
the experiences important to the participants, some unexpected. 
These experiences illuminate the strengths and weaknesses of the 
Web site, and our knowledge of them will facilitate future decisions 
intended to maximize its potential to affect capacity building in 
evaluation. We believe this study will initiate further research 
into the use of a Web site for capacity building to achieve orga¬ 
nizational change. 

In conclusion, our research suggests that a Web site can 
achieve individual and collective learning in evaluation, leading 
to an emerging culture of evaluation in an organization. To meet 
these expectations, however, the creation of a Web site should be 
approached as an intentional scholarly activity that is informed 
by theory and driven by organizational leadership. 
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